Rendering device, head-mounted display, image transmission method, and image correction method

ABSTRACT

Provided is a head-mounted display  100  for displaying images transmitted from a rendering device  200 . An orientation sensor  64  detects an orientation of the head-mounted display  100 . An HDMI transmission/reception unit  90  receives frame data of an image to be displayed on the head-mounted display  100  and predicted orientation information of the head-mounted display  100  that is transmitted synchronously with the frame data of the image. A re-projection unit  60  corrects the frame data of the image on the basis of a difference between a latest orientation information detected by the orientation sensor  64  and the predicted orientation information so that the frame data of the image suits a latest orientation of the head-mounted display  100.

TECHNICAL FIELD

The present invention relates to an image transmission technique and animage correction technique for a head-mounted display.

BACKGROUND ART

Playing a game is popularly enjoyed by operating a controller or thelike while wearing a head-mounted display connected to a game machine onthe head and viewing an image displayed on the head-mounted display.When the head-mounted display is attached, the user sees only the videodisplayed on the head-mounted display, so that the sense of immersion inthe video world is enhanced and thus the entertainment quality of thegame is further improved. In addition, if it is configured that virtualreality video is displayed on the head-mounted display and a 360 degreeview of an entire circumferential virtual space can be displayed whenthe user wearing the head-mounted display turns the user's head around,the sense of immersion in the video is further enhanced and theoperability of applications such as games is further improved.

SUMMARY Technical Problem

In this way, when a head-mounted display is provided with a headtracking function and virtual reality video is generated by changing theviewpoint and the visual line direction in conjunction with the movementof the user's head, since a time lag exists from the generation of thevirtual reality video until the display thereof, difference occursbetween the user's head orientation used as a presupposition at the timeof the video generation and the user's head orientation at the time thevideo is displayed on the head-mounted display, and the user may startto feel motion sickness (called “virtual reality (VR) sickness”).

The present invention has been made in view of these problems, and anobject thereof is to provide an image transmission technique and animage correction technique for coping with the VR sickness from ahead-mounted display.

Solution to Problem

In order to solve the above problems, a rendering device according to anaspect of the present invention includes: a prediction unit thatpredicts an orientation of a head-mounted display after delay time haselapsed based on orientation information received from the head-mounteddisplay, and outputs predicted orientation information; an imagegeneration unit that generates an image to be displayed on thehead-mounted display based on the predicted orientation information ofthe head-mounted display; and a transmission unit that transmits thepredicted orientation information of the head-mounted display to thehead-mounted display in synchronization with frame data of the image.

Another aspect of the present invention is a head-mounted display. Thishead-mounted display is one for displaying an image transmitted from arendering device and includes: a sensor that detects orientationinformation of the head-mounted display; a reception unit that receivesframe data of an image to be displayed on the head-mounted display andpredicted orientation information of the head-mounted displaytransmitted in synchronization with the frame data of the image; and are-projection unit that corrects the frame data of the image so that theframe data suits a latest orientation of the head-mounted display basedon a difference between latest orientation information detected by thesensor and the predicted orientation information.

Yet another aspect of the present invention is an image transmissionmethod. This method includes: a prediction step of predicting anorientation of a head-mounted display after delay time has elapsed basedon orientation information received from the head-mounted display tooutput predicted orientation information; an image generation step ofgenerating an image to be displayed on the head-mounted display based onthe predicted orientation information of the head-mounted display; and atransmission step of transmitting the predicted orientation informationof the head-mounted display to the head-mounted display insynchronization with frame data of the image.

Further, another aspect of the present invention is an image correctionmethod. This method is one for correcting an image transmitted from arendering device and to be displayed on a head-mounted display andincludes: a reception step of receiving frame data of an image to bedisplayed on the head-mounted display and predicted orientationinformation of the head-mounted display transmitted in synchronizationwith the frame data of the image; a detection step of detecting latestorientation information of the head-mounted display; and a re-projectionstep of correcting the frame data of the image so that the frame datasuits a latest orientation of the head-mounted display based on adifference between the latest orientation information and the predictedorientation information.

Still further, another aspect of the present invention is also an imagecorrection method. This method is one for performing re-projection on animage by transmitting the image from a rendering device to ahead-mounted display and includes: a transmission step of transmittinginformation used as a presupposition when the rendering device draws theimage to be displayed on the head-mounted display and necessary forre-projection in the head-mounted display to the head-mounted display insynchronization with frame data of the image; and a re-projection stepof correcting the frame data of the image by the head-mounted display byapplying information that is transmitted in synchronization with theframe data of the image and is necessary for the re-projection to theframe data of the image.

It should be noted that any combination of the above-describedconstituent elements and the expression of the present inventionconverted between a method, device, system, computer program, datastructure, recording medium, etc. are also effective as an aspect of thepresent invention.

Advantageous Effect of Invention

According to the present invention, it is possible to cope with the VRsickness of a head-mounted display.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an external view of a head-mounted display.

FIG. 2 is a configuration diagram of an image transfer system accordingto the present embodiment.

FIG. 3 is a functional configuration diagram of the head-mounteddisplay.

FIG. 4 is a functional configuration diagram of a rendering deviceaccording to the present embodiment.

FIG. 5 is a diagram illustrating an entire circumferential imagedisplayed on the head-mounted display.

FIG. 6 is a diagram for explaining the reason why a delay occurs in theentire circumferential image displayed on the head-mounted display.

FIG. 7 is a diagram illustrating a method for transmitting dynamicmetadata through a transmission path different from that of frame data.

FIG. 8 is a diagram illustrating a method for transmitting dynamicmetadata through the same transmission path in synchronization withframe data.

FIG. 9 is a sequence diagram illustrating re-projection processing bythe head-mounted display and the rendering device.

FIG. 10 is a diagram illustrating divided areas of multi-resolutionshading.

FIGS. 11(a) to 11(c) are diagrams illustrating how the size of thedivided area of the multi-resolution shading changes according to theload of a graphics processing unit (GPU).

FIG. 12 is a diagram illustrating a gaze area in foveated rendering.

FIGS. 13A to 13C are diagrams illustrating how the position of the gazearea of the foveated rendering changes according to the movement of agaze point.

FIGS. 14A to 14C are diagrams illustrating how the position of a dividedarea changes when gaze tracking is combined with the multi-resolutionshading.

DESCRIPTION OF EMBODIMENT

FIG. 1 is an external view of a head-mounted display 100. Thehead-mounted display 100 is a display device to be used for enjoyingstill images and moving images displayed on the display and forlistening to audio and music output from headphones, while being worn onthe user's head.

The position information of the head of the user wearing thehead-mounted display 100 and the orientation information such as therotation angle and inclination of the head can be measured by a gyrosensor or an acceleration sensor built in or externally attached to thehead-mounted display 100.

The head-mounted display 100 may further be provided with a camera thatcaptures an image of the user's eyes. The camera mounted on thehead-mounted display 100 can detect the user's gaze direction, pupilmovement, blinking, and the like.

The head-mounted display 100 is an example of a “wearable display”.Here, a method for generating an image displayed on the head-mounteddisplay 100 will be described, but the image generating method accordingto the present embodiment can also be applied to the cases whereglasses, a glasses type display, a glasses type camera, headphones, aheadset (headphones with a microphone), earphones, earrings, anear-mounted camera, a cap, a cap with a camera, a hair band, and thelike are worn, without being limited to the head-mounted display 100 ina narrow sense.

FIG. 2 is a configuration diagram of the image transfer system accordingto the present embodiment. As an example, the head-mounted display 100is connected to a rendering device 200 through an interface such as ahigh-definition multimedia interface (HDMI) (registered trademark) whichis a standard of a communication interface for transmitting video andaudio as digital signals.

In the present embodiment, a data transmission path 300 between thehead-mounted display 100 and the rendering device 200 is an HDMItransmission path.

The HDMI 2.1 standard has a function called a high dynamic range (HDR),which refers to the dynamic metadata of video and can generate video forwhich the luminance and color depth are adjusted optimally for eachframe according to the scene. In the HDMI 2.1 standard, dynamic metadatacan transmit information necessary for a dynamic HDR such as maximumluminance, average luminance, and minimum luminance of a scene insynchronization with video.

In the present embodiment, the rendering device 200 predicts theposition-orientation information of the head-mounted display 100 inconsideration of the delay time from the generation of video to thedisplay thereof, and causes the predicted position-orientationinformation of the head-mounted display 100 used as a presupposition atthe time of making an image to be included in the dynamic metadata,thereby transmitting the dynamic metadata to the head-mounted display100 in synchronization with the video frame data.

The communication interface between the head-mounted display 100 and therendering device 200 is not limited to the HDMI as long as dynamicmetadata can be transmitted in synchronization with video.

An example of the rendering device 200 is a game machine. The renderingdevice 200 may be further connected to a server via a network. In thatcase, the server may provide the rendering device 200 with an onlineapplication such as a game in which a plurality of users can participatevia a network. The head-mounted display 100 may be connected to acomputer or a portable terminal instead of the rendering device 200.

The video displayed on the head-mounted display 100 may be video basedon computer graphics such as a game video in addition to video imagespreviously captured by a camera. Further, the video may be live video ofa remote place distributed via a network.

FIG. 3 is a functional configuration diagram of the head-mounted display100.

A control unit 10 is a main processor that processes and outputs signalssuch as image signals and sensor signals, commands, and data. An inputinterface 20 receives operation signals and setting signals from theuser and supplies the signals to the control unit 10. An outputinterface 30 receives the image signal from the control unit 10 anddisplays the image on the display. A backlight 32 supplies backlight tothe liquid crystal display.

A communication control unit 40 transmits data input from the controlunit 10 to the outside through wired or wireless communication via anetwork adapter 42 or an antenna 44. The communication control unit 40also receives data from the outside through wired or wirelesscommunication via the network adapter 42 or the antenna 44 and outputsthe data to the control unit 10.

A storage unit 50 temporarily stores data, parameters, operationsignals, and the like processed by the control unit 10.

An external input/output terminal interface 70 is an interface forconnecting peripheral equipment such as a universal serial bus (USB)controller. An external memory 72 is an external memory such as a flashmemory.

A clock unit 80 sets time information according to a setting signal fromthe control unit 10 and supplies time data to the control unit 10.

An HDMI transmission/reception unit 90 transmits or receives video-audiodigital signals according to the HDMI. Dynamic metadata is associatedwith the frame data received by the HDMI transmission/reception unit 90from the rendering device 200, and the dynamic metadata includespredicted position-orientation information L2 of the head-mounteddisplay 100 that the rendering device 200 uses as a presupposition whenmaking an image by the frame data.

The control unit 10 can supply the image and text data to the outputinterface 30 for display thereof on the display or to the communicationcontrol unit 40 for transmission thereof to the outside.

An orientation sensor 64 detects position information of thehead-mounted display 100 and orientation information such as therotation angle and inclination of the head-mounted display 100. Theorientation sensor 64 is obtained by appropriately combining a gyrosensor, an acceleration sensor, an angular acceleration sensor, and thelike. A back-and-forth, right-and-left and up-and-down movement of theuser's head may be detected by using a motion sensor obtained bycombining at least one of a three-axis geomagnetic sensor, a three-axisacceleration sensor, and a three-axis gyro (angular velocity) sensor.

The rendering device 200 is notified of current position-orientationinformation L1 of the head-mounted display 100 detected by theorientation sensor 64 via the communication control unit 40 or theexternal input/output terminal interface 70. Alternatively, the HDMItransmission/reception unit 90 may transmit the currentposition-orientation information L1 of the head-mounted display 100 tothe rendering device 200. The rendering device 200 predicts theposition-orientation information of the head-mounted display 100 fromthe received current position-orientation information L1 of thehead-mounted display 100 in consideration of the delay time from thegeneration of the video to the display thereof, and draws the image tobe displayed on the head-mounted display 100 using the predictedposition-orientation information L2 of the head-mounted display 100 as apresupposition.

At the time of reception of the image drawn data from the renderingdevice 200, the orientation sensor 64 detects latestposition-orientation information L3 of the head-mounted display 100, andgives the information to a position-orientation difference calculationunit 62. The position-orientation difference calculation unit 62receives the predicted position-orientation information L2 used as apresupposition when the rendering device 200 draws an image from theHDMI transmission/reception unit 90. The position-orientation differencecalculation unit 62 calculates a difference ΔL between the latestposition-orientation information L3 and the predictedposition-orientation information L2, and gives the difference ΔL to are-projection unit 60. Here, note that as for the difference ΔL betweenthe latest position-orientation information L3 and the predictedposition-orientation information L2, these pieces of information aregenerally different in both the position information and the orientationinformation of the head-mounted display 100, but only one of theposition information and the orientation information may be different.

The re-projection unit 60 executes re-projection by applying correctionbased on the difference ΔL to the image drawn data received by the HDMItransmission/reception unit 90 from the rendering device 200, and givesthe re-projected image drawn data to the control unit 10. The controlunit 10 supplies the re-projected image drawn data to the outputinterface 30 for displaying the data on the display.

FIG. 4 is a functional configuration diagram of the rendering device 200according to the present embodiment. This figure illustrates a blockdiagram focusing on functions, and these functional blocks can berealized in various forms by only hardware, only software, or acombination thereof.

At least a part of the functions of the rendering device 200 may bemounted on the head-mounted display 100. Alternatively, at least a partof the functions of the rendering device 200 may be mounted in a serverconnected to the rendering device 200 via a network.

A position-orientation acquisition unit 210 acquires the currentposition-orientation information L1 of the head-mounted display 100 fromthe head-mounted display 100.

A delay time acquisition unit 220 acquires a delay time required frommaking an image that is visible in the visual line direction from aviewpoint position at a certain time until display of the image on thehead-mounted display 100. This delay time includes not only the timerequired for the image drawn process but also the time required totransmit the image data. The delay time acquisition unit 220 obtains adelay time based on the three-dimensional image drawn hardwareperformance and the transmission delay of the transmission path.

A position-orientation prediction unit 230 predicts the amount of changein position and orientation during the delay time obtained by the delaytime acquisition unit 220. The position-orientation prediction unit 230can determine the amount of change in the position and orientation bymultiplying the translational velocity and angular velocity of the headof the user wearing the head-mounted display 100 by the delay time. Theposition-orientation prediction unit 230 predicts theposition-orientation information L2 after the lapse of the delay time byadding the change amount of the position and orientation during thedelay time to the current position-orientation information L1, andsupplies the predicted position-orientation information L2 to aviewpoint-visual line setting unit 240 and a metadata generation unit270.

The viewpoint-visual line setting unit 240 updates the user's viewpointposition and visual line direction using the predictedposition-orientation information L2 of the head-mounted display 100acquired by the position-orientation prediction unit 230.

An image generation unit 250 reads the image data from an image storageunit 260, and generates an image seen in the visual line direction fromthe viewpoint position of the user wearing the head-mounted display 100according to the user's viewpoint position and visual line direction setby the viewpoint-visual line setting unit 240, and gives the image to anHDMI transmission/reception unit 280. Here, the image data may be amoving image or a still image content created in advance, or may becomputer graphics subjected to rendering.

The metadata generation unit 270 acquires the predictedposition-orientation information L2 from the position-orientationprediction unit 230, embeds the predicted position-orientationinformation L2 in the dynamic metadata to be associated with the framedata, and supplies the dynamic metadata to the HDMItransmission/reception unit 280.

The HDMI transmission/reception unit 280 receives frame data from theimage generation unit 250 and receives dynamic metadata in which thepredicted position-orientation information L2 is embedded from themetadata generation unit 270. The HDMI transmission/reception unit 280synchronizes the dynamic metadata with the frame data according to theHDMI, and transmits the frame data and the dynamic metadata to thehead-mounted display 100.

FIG. 5 is a diagram illustrating an entire circumferential image 500displayed on the head-mounted display 100. When the user is facingtoward the left front with respect to the entire circumferential image500, an image 510 a located in the range of an angle of view 150 a inthe direction of a head-mounted display 100 a is displayed, and when theuser is facing toward the right front after turning his/her neck, animage 510 b located in the range of an angle of view 150 b in thedirection of a head-mounted display 100 b is displayed.

As described above, since the viewpoint position and the visual linedirection for viewing the entire circumferential image displayed on thehead-mounted display 100 change according to the movement of the head,the feeling of immersion in the entire circumferential image can beenhanced.

FIG. 6 is a diagram for explaining the reason why the entirecircumferential image displayed on the head-mounted display 100 isdelayed. When the user turns the user's neck and faces toward the rightfront, the image 510 b located in the range of the angle of view 150 bin the direction of the head-mounted display 100 b is generated anddisplayed on the head-mounted display 100, but at the time of displayingthe image 510 b, the position and rotation of the head-mounted display100 b have already changed as indicated by reference sign 150 c. Forthis reason, although it is necessary to display an image that issupposed to be visible in the range of the angle of view 150 c on ahead-mounted display 100 c, the image actually generated and displayedis an image that is seen in the range of the angle of view 150 b in thedirection of the head-mounted display 100 b, which is an image at aslightly previous time. Due to a deviation from this time lag, an imageslightly deviated from one in the direction in which the user is lookingis displayed on the head-mounted display 100, and the user may feel somekind of “motion sickness”.

In this way, since after the rotation of the head-mounted display 100 isdetected, the next image drawn range is determined, and the CPU issuesan drawn generation command, and then a GPU executes rendering, it takestime before outputting a generated image to the head-mounted display100. If image drawing is performed at a frame rate of 60 fps(frames/second), for example, even if the CPU operates at sufficientlyhigh speed, delay time corresponding to one frame has been caused whenthe image is output after the rotation of the head-mounted display 100is detected. This is approximately 16.67 milliseconds under a frame rateof 60 fps, which is a sufficient time for humans to detect a deviation.

Further, latency occurs when an image drawn by the rendering device 200is transmitted to the head-mounted display 100 via the data transmissionpath 300.

Therefore, a re-projection process is performed on the generated imageto make it difficult for humans to detect the deviation. Although imagedrawing of the rendering device 200 performed by predicting theposition-orientation information of the head-mounted display 100 afterthe delay time has elapsed is one kind of re-projection, imagecorrection in order to compensate for the deviation of the predictedposition-orientation information of the head-mounted display 100 whichis used as a presupposition when the rendering device 200 draws an imagefrom the latest position-orientation information of the head-mounteddisplay 100 when the image drawn by the rendering device 200 isdisplayed on the head-mounted display 100 is called re-projection in thepresent embodiment.

To be specific, this is a process of determining the difference betweenthe predicted position-orientation information of the head-mounteddisplay 100 used as a presupposition when image drawing is performed andthe latest position-orientation information of the head-mounted display100, and correcting an image so that the image suits the latestposition-orientation information of the head-mounted display 100, andtechniques such as image conversion and frame interpolation are used.

The rendering device 200 transmits information necessary forre-projection to the head-mounted display 100 in synchronization withimage frame data as dynamic metadata. Since the rendering device 200 hasthe predicted position-orientation information of the head-mounteddisplay 100 used as a presupposition when making an image, withouttransmission of the information as dynamic metadata from the renderingdevice 200 to the head-mounted display 100 for each frame, theinformation could not be known by the head-mounted display 100.Therefore, the rendering device 200 embeds the predictedposition-orientation information of the head-mounted display 100 in thedynamic metadata, and transmits the dynamic metadata to the head-mounteddisplay 100 in synchronization with the frame data. The predictedposition-orientation information of the head-mounted display 100 is anexample of data necessary for re-projection, and other information maybe embedded in dynamic metadata and transmitted to the head-mounteddisplay 100 if the information is necessary for re-projection.

Here, if dynamic metadata is transmitted separately from frame data, theframe data could not be associated with the dynamic metadata, and thedynamic metadata could not be applied to the frame data. For comparison,referring to FIG. 7, a method for transmitting dynamic metadataseparately without associating the metadata with frame data will bedescribed, and then, referring to FIG. 8, a method for transmitting thedynamic metadata while the data is synchronized with frame data will bedescribed.

FIG. 7 is a diagram for describing a method for transmitting dynamicmetadata through a transmission path different from that of frame data.

Frame data is transmitted through the HDMI transmission path. A verticalblanking signal VBlank is inserted immediately before each frame data.Here, a signal VBlank 602 is inserted immediately before Nth frame data600, and a signal VBlank 612 is inserted immediately before (N+1)thframe data 610.

As an example, dynamic metadata to be applied to each frame data istransmitted through a USB transmission path. In addition to USBconnection, metadata may be transmitted by wireless communication, forexample. Nth dynamic metadata 604 to be applied to the Nth frame data600 and (N+1)th dynamic metadata 614 to be applied to the (N+1)th framedata 610 are transmitted through the USB transmission path, but thesepieces of the dynamic metadata 604 and 614 are not synchronized with theframe data 600 and 610. For this reason, even if the head-mounteddisplay 100 receives the dynamic metadata 604 and 614, whether themetadata should be applied to the frame data 600 or 610 is unknown.

FIG. 8 is a diagram for describing a method for transmitting dynamicmetadata through the same transmission path in synchronization withframe data.

Frame data and dynamic metadata are transmitted through the HDMItransmission path. A vertical blanking signal VBlank is insertedimmediately before each frame data, and metadata is inserted into thevertical blanking signal VBlank. Here, the signal VBlank 602 is insertedimmediately before the Nth frame data 600, and the Nth dynamic metadata604 is inserted into the signal VBlank 602. Also, the signal VBlank 612is inserted immediately before the (N+1)th frame data 610, and the(N+1)th dynamic metadata 614 is inserted into the signal VBlank 612.

As described above, when the dynamic metadata 604 and 614 to be appliedto the frame data 600 and 610 are inserted into the vertical blankingsignals VBlank of the frame data 600 and 610 respectively andtransmitted through the same HDMI transmission path, since the dynamicmetadata 604 and 614 are synchronized with the frame data 600 and 610respectively, the head-mounted display 100 can correctly apply thedynamic metadata 604 and 614 to the corresponding frame data 600 and 610respectively.

In the HDMI 2.1 standard, it is studied to insert dynamic metadata to beapplied to each frame in the VBlank signal of each frame data totransmit the metadata. When the predicted position-orientationinformation of the head-mounted display 100 is embedded in the dynamicmetadata, the predicted position-orientation information of thehead-mounted display 100 used by the rendering device 200 can betransmitted to the head-mounted display 100 in synchronization with theframe data. Here, an example has been described in which dynamicmetadata is inserted into an HDMI 2.1 standard VBlank signal, but thisis only an example, and dynamic metadata may be inserted into somesynchronization signal synchronized with each frame and transmitted.

FIG. 9 is a sequence diagram for describing the re-projection processperformed by the head-mounted display 100 and the rendering device 200.

The orientation sensor 64 of the head-mounted display 100 detects thecurrent position-orientation information L1 of the head-mounted display100 (S10). The head-mounted display 100 notifies the rendering device200 of the current position-orientation information L1 (S12).

The rendering device 200 determines a delay time generated by atransmission delay between the head-mounted display 100 and therendering device 200 and a processing delay for drawing image in therendering device 200. The position-orientation prediction unit 230 ofthe rendering device 200 predicts the position-orientation informationL2 of the head-mounted display 100 after the delay time has elapsed fromthe current position-orientation information L1 (S14).

The image generation unit 250 of the rendering device 200 draws an imagethat is visible to the user wearing the head-mounted display 100, basedon the predicted position-orientation information L2 (S16).

The HDMI transmission/reception unit 280 of the rendering device 200associates the predicted position-orientation information L2 as dynamicmetadata with the frame data in which an image has been drawn andtransmits the data to the head-mounted display 100 (S18).

When the frame data is received from the rendering device 200, theorientation sensor 64 of the head-mounted display 100 detects the latestposition-orientation information L3 of the head-mounted display 100, andthe position-orientation difference calculation unit 62 determines thedifference ΔL between the latest position-orientation information L3 andthe predicted position-orientation information L2 (S20).

The re-projection unit 60 performs re-projection processing on the framedata based on the difference ΔL, and generates an image that suits thelatest position-orientation information L3 (S22).

Although the predicted position-orientation information L2 is embeddedin the dynamic metadata and transmitted in synchronization with theframe data according to the above description, information necessary forre-projection is embedded in the dynamic metadata and transmitted ingeneral. Hereinafter, other information to be embedded in the dynamicmetadata will be described with reference to embodiment usingre-projection.

FIG. 10 is a diagram for illustrating divided areas of multi-resolutionshading. When video is displayed on the head-mounted display 100, it isnecessary to correct the lens distortion of the image, and theresolution is maintained in the central area of the image, but the areaat the end of the visual field is compressed and the resolution islowered. Therefore, there is a method called the multi-resolutionshading that performs image drawing in a central area 400 of the imagewith high resolution and in the other peripheral areas with lowresolution to reduce the load of the image drawn process. In thehead-mounted display 100, although the multi-resolution shading isperformed on each left-eye image and right-eye image, here the image ofone eye is illustrated.

The rendering device 200 transmits an image drawn by themulti-resolution shading to the head-mounted display 100, and there-projection unit 60 of the head-mounted display 100 executesre-projection on the image in accordance with the latestposition-orientation information L3. When executing re-projection on animage, the head-mounted display 100 needs to know how the image has beendivided to be drawn at a plurality of resolutions in themulti-resolution shading. This is because the difference in resolutionfor each divided area must be taken into account when re-projecting theimage.

Therefore, the rendering device 200 causes information associated withthe divided areas in the multi-resolution shading to be included in thedynamic metadata in addition to the predicted position-orientationinformation L2, and transmits the information to the head-mounteddisplay 100 in synchronization with the frame data.

FIGS. 11(a) to 11(c) are diagrams for describing how the size of thedivided area for the multi-resolution shading changes according to theload of the GPU. When the GPU load is small, a central area where animage is drawn with a high resolution is widened (reference sign 402) asillustrated in FIG. 11(a), but as the GPU load increases, the centralarea 402 is narrowed (reference signs 404 and 406) as illustrated FIGS.11(b) and 11(c). As described above, when the size of the divided areachanges according to the load of the GPU, the range information of thedynamically changing divided area is included in the dynamic metadatafor each frame and transmitted to the head-mounted display 100.

FIG. 12 is a diagram illustrating a gaze area in foveated rendering.There is a method called the foveated rendering, which performs theuser's gaze tracking, draws an image with high resolution in the areahaving the gaze point as the center, and draws an image in thesurroundings with low resolution to reduce the image drawn processingload. An image is drawn with high resolution in a central area 410having the gaze point as the center, with medium resolution in aperipheral area 420 outside the central area 410, and with lowresolution on the outer side of the peripheral area 420.

The rendering device 200 transmits an image drawn by the foveatedrendering to the head-mounted display 100, and the re-projection unit 60of the head-mounted display 100 executes re-projection on the image inaccordance with the latest position-orientation information L3. When thehead-mounted display 100 executes re-projection on an image, informationassociated with the gaze area in the foveated rendering is necessary.Therefore, the rendering device 200 causes information associated withthe gaze area in the foveated rendering, specifically informationassociated with the position of the gaze point and the radius of thecircle of the gaze area, to be included in the dynamic metadata, inaddition to the predicted position-orientation information L2, andtransmits the information to the head-mounted display 100 insynchronization with the frame data.

FIGS. 13A to 13C are diagrams illustrating how the position of the gazearea in the foveated rendering changes in accordance with the movementof the gaze point. When the gaze point is in the lower left, the gazearea is in the lower left (central area 412 and peripheral area 422) asillustrated in FIG. 13A, and when the gaze point is in the front, thegaze area is in the front (central area 414, peripheral area 424) asillustrated in FIG. 13B, and when the gaze point is in the upper right,the gaze area is in the upper right (central area 416, peripheral area426) as illustrated in FIG. 13C. As described above, when the positionof the gaze area changes according to the position of the gaze point,the position information of the gaze area that dynamically changes isincluded in the dynamic metadata for each frame and transmitted to thehead-mounted display 100.

As in FIGS. 11(a) to 11(c), when the size of the gaze area in thefoveated rendering changes according to the GPU load, that is, in thecase where the gaze area is widened when the GPU load is small and thegaze area is narrowed as the load increases, the range information ofthe gaze area that dynamically changes is included in the dynamicmetadata for each frame and transmitted to the head-mounted display 100.

FIGS. 14A to 14C are diagrams illustrating how the position of thedivided area changes when gaze tracking is combined with themulti-resolution shading. When the gaze point is in the lower left, afixation area 432 where an image is drawn with high resolution is in thelower left as illustrated in FIG. 14A, and when the gaze point is infront, a fixation area 434 is in the front as illustrated in FIG. 14B,and further when the gaze point is in the upper right, a fixation area436 is in the upper right as illustrated in FIG. 14C. Thus, when theposition of the fixation area changes according to the position of thegaze point, the position information of the fixation area thatdynamically changes is included in the dynamic metadata for each frameand transmitted to the head-mounted display 100.

As described above, according to the rendering device 200 of the presentembodiment, information necessary for re-projection can be transmittedas dynamic metadata to the head-mounted display 100 in synchronizationwith image frame data. The head-mounted display 100 can extractinformation necessary for re-projection from dynamic metadata to applyre-projection to the image frame data. Thereby, VR sickness of the userwearing the head-mounted display 100 can be prevented.

The present invention has been described based on the embodiment. Theembodiment is an example, and it will be understood by those skilled inthe art that various modifications can be made to combinations of therespective constituent elements and processing processes, and suchmodifications are within the scope of the present invention. Suchmodifications are described.

REFERENCE SIGNS LIST

10 Control unit, 20 Input interface, 30 Output interface 32 Backlight,40 Communication control unit, 42 Network adapter, 44 Antenna, 50Storage unit, 60 Re-projection unit, 62 Position-orientation differencecalculation unit, 64 Orientation sensor, 70 External input/outputterminal interface, 72 External memory, 80 Clock unit, 90 HDMItransmission/reception unit, 100 Head-mounted display, 200 Renderingdevice, 210 Position-orientation acquisition unit, 220 Delay timeacquisition unit, 230 Position-orientation prediction unit, 240Viewpoint-visual line setting unit, 250 Image generation unit, 260 Imagestorage unit, 270 Metadata generation unit, 280 HDMItransmission/reception unit, 300 Data transmission path, 500 Entirecircumferential image.

INDUSTRIAL APPLICABILITY

The present invention can be used for image transmission technology andimage correction technology for head-mounted displays.

The invention claimed is:
 1. A rendering device wirelessly coupled to ahead-mounted display, the rendering device comprising: a prediction unitthat predicts an orientation of the head-mounted display after delaytime has elapsed based on orientation information received wirelesslyfrom the head-mounted display, and outputs predicted orientationinformation; an image generation unit that generates an image to bedisplayed on the head-mounted display based on the predicted orientationinformation of the head-mounted display; and a transmission unit thattransmits the predicted orientation information of the head-mounteddisplay to the head-mounted display in synchronization with frame dataof the image, wherein the head-mounted display uses the predictedorientation information to re-project the frame data of the image. 2.The rendering device according to claim 1, wherein the transmission unitinserts the predicted orientation information of the head-mounteddisplay into a synchronization signal preceding each frame as metadataand transmits the predicted orientation information.
 3. A head-mounteddisplay for displaying an image wirelessly transmitted from a renderingdevice, the head-mounted display comprising: a sensor that detectsorientation information of the head-mounted display; a reception unitthat receives frame data of an image to be displayed on the head-mounteddisplay and predicted orientation information of the head-mounteddisplay transmitted in synchronization with the frame data of the image,wherein the predicted orientation information is calculated by therendering device; and a re-projection unit that corrects the frame dataof the image such that the frame data suits a latest orientation of thehead-mounted display based on a difference between latest orientationinformation detected by the sensor and the predicted orientationinformation.
 4. The head-mounted display according to claim 3, whereinthe reception unit receives the transmitted predicted orientationinformation that has been inserted into a synchronization signal of eachframe as metadata.
 5. The head-mounted display according to claim 4,wherein the metadata includes information associated with a position orsize of a divided area of multi-resolution shading that dynamicallychanges according to a processing load or movement of a gaze point, andthe re-projection unit corrects the frame data of the image withreference to the information associated with the position or the size ofthe divided area of the multi-resolution shading.
 6. The head-mounteddisplay according to claim 4, wherein the metadata includes informationassociated with a position or the size of a gaze area of foveatedrendering that dynamically changes according to a processing load ormovement of a gaze point, and the re-projection unit corrects the framedata of the image with reference to the information associated with theposition or the size of the gaze area of the foveated rendering.
 7. Animage correction method for correcting an image that is wirelesslytransmitted from a rendering device and is to be displayed on ahead-mounted display, the method comprising: receiving frame data of animage to be displayed on the head-mounted display and predictedorientation information of the head-mounted display transmitted insynchronization with the frame data of the image, wherein the predictedorientation information is calculated by the rendering device; detectinglatest orientation information of the head-mounted display; andcorrecting the frame data of the image such that the frame data suits alatest orientation of the head-mounted display based on a differencebetween the latest orientation information calculated by thehead-mounted display and the predicted orientation informationcalculated by the rendering device.